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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Program  Executive  Officer,  SHIPS 
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Software  Should-Cost  Analysis  With  Parametric 

Estimation  T ools1 

Robert  Ferguson — Ferguson  is  a  senior  member  of  the  technical  staff  at  the  Software  Engineering 
Institute  (SEI).  He  works  primarily  on  software  measurement  and  estimation.  He  spent  30  years  in  the 
industry  as  a  software  developer  and  project  manager  before  coming  to  the  SEI.  His  experience 
includes  applications  in  real-time  flight  controls,  manufacturing  control  systems,  large  databases,  and 
systems  integration  projects.  He  has  also  frequently  led  process  improvement  teams.  Ferguson  is  a 
senior  member  of  IEEE  and  has  a  Project  Management  Professional  (PMP)  certification  from  the 
Project  Management  Institute  (PMI).  [rwf@sei.cmu.edu] 

Abstract 

In  Department  of  Defense  (DoD)  acquisition  contracts  there  are  often  concerns  of  security 
and  competitive  advantage  making  it  difficult  to  find  comparable  performance  data  that  may 
be  useful  in  evaluating  contractor  proposals.  In  order  for  programs  to  make  such  comparative 
evaluations,  a  should-cost  analysis  may  be  conducted.  This  analysis  can  be  compared  to  a 
benchmarking  process  provided  that  a  benchmark  database  is  available.  Parametric 
estimation  tools  provide  this  type  of  data. 

This  paper  shows  how  SEER-SEM  was  applied  as  part  of  the  should-cost  effort  on  the  F-22 
program.  The  Office  of  the  Secretary  of  Defense  recognized  the  resulting  $32  million  savings 
in  the  presentation  on  Better  Buying  Power  II. 

Introduction 

June  28,  2010,  Under  Secretary  Ashton  Carter  issued  the  Better  Buying  Power 
memorandum  (Carter,  2010)  suggesting  seven  (7)  focus  topics.  “Should-cost  analysis” 
addresses  several  of  the  focus  areas  but  most  clearly  the  one  Secretary  Gates  labeled 
“Incentivize  Productivity  and  Innovation  in  Industry  and  Government.”  The  Department  of 
Defense  (DoD)  has  significant  history  with  should-cost  analyses.  A  RAND  study  (Boito, 
2012),  examined  this  history  from  the  1970s  to  today.  The  RAND  study  finds  support  for  this 
analysis  in  the  Federal  Acquisition  Regulations  (FAR)  as  follows: 

Should-cost  analysis  as  described  in  the  FAR  is  a  specialized  form  of  cost 
analysis,  used  to  support  contract  negotiations,  that  is  characterized  by  a 
focus  on  the  elimination  of  contractor  inefficiencies.  It  is  significant  that  the 
guidance  for  should  cost  analysis  is  found  in  the  federal  regulation  for  the 
contracting  function,  because  contracting  is  the  process  by  which  the 
government  specifies  what  it  wants  to  buy  and  at  what  price.  (Boito,  2012,  p. 
41) 


1  Copyright  2013  Carnegie  Mellon  University 

This  material  is  based  upon  work  funded  and  supported  by  the  Department  of  Defense  under 
Contract  No.  FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software 
Engineering  Institute,  a  federally  funded  research  and  development  center. 

NO  WARRANTY.  THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING 
INSTITUTE  MATERIAL  IS  FURNISHED  ON  AN  “AS-IS”  BASIS.  CARNEGIE  MELLON  UNIVERSITY 
MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO  ANY 
MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR  PURPOSE  OR 
MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM  USE  OF  THE  MATERIAL. 
CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY  WARRANTY  OF  ANY  KIND  WITH 
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In  this  study,  RAND  observes  that  a  should-cost  analysis  requires  participation  of 
both  contractors  and  government  personnel.  Successful  negotiation  can  only  be  achieved 
when  the  contractor  agrees  to  the  objectivity  of  government  observations  and  the  contractor 
believes  it  can  eliminate  the  inefficiency.  The  negotiation  task  is  often  difficult  because  the 
government  is  frequently  in  a  position  of  having  a  single  source  supplier.  The  single-source 
situation  may  make  it  difficult  for  the  government  to  persuade  the  contractor  to  participate 
openly  in  the  should-cost  analysis.  Any  lack  of  openness  or  access  to  data  will  limit  the 
government’s  ability  to  identify  the  inefficiencies. 

A  major  challenge  in  conducting  a  should-cost  analysis  is  the  skill  required  of  the 
analysts.  The  team  doing  the  analysis  must  encompass  skills  in  pricing,  contracting, 
program  management,  and  subject  matter  expertise  in  areas  relevant  to  the  program  (Boito, 
2012,  p.  x).  This  team  must  have  both  depth  of  knowledge  in  the  focus  disciplines  and 
breadth  of  experience  across  programs  and  industry.  Finally,  they  must  be  able  to  apply 
these  skills  to  present  an  objective  set  of  recommendations  accessible  to  both  program 
management  and  contractor. 

The  Software  Engineering  Institute  (SEI)  has  participated  in  some  should-cost 
analyses  using  parametric  software  cost  estimation  tools.  This  paper  describes  the 
methodology  and  some  results.  The  following  section  describes  the  methodology.  Then 
next  section  discusses  an  example  application  and  results  synthesized  from  multiple  cases. 
The  final  section  provides  lessons  learned  and  ideas  for  future  improvements. 

SEI  Should-Cost  Methodology 

The  DoD  may  have  gotten  an  early  start  on  everyone  with  “should-cost  analysis,”  but 
the  commercial  world  has  pursued  the  topic  extensively  under  the  label  of  “benchmarking.” 
An  early  book  on  the  subject  is  Benchmarking:  The  Search  for  Industry  Best  Practices  That 
Lead  to  Superior  Performance  by  R.C.  Camp  (Camp,  1989).  Just  a  year  later,  James 
Womack,  Daniel  T.  Jones,  and  Daniel  Roos  (1990)  described  Toyota’s  use  of  benchmarking 
in  The  Machine  That  Changed  the  World.  In  the  1990s,  corporate  benchmarking  was  a 
popular  consulting  business. 

The  SEI  should-cost  work  stemmed  directly  from  SEI  experience  with  benchmark 
databases  in  the  form  of  parametric  cost  estimation  tools.  Using  the  parametric  estimation 
tools  is  not  quite  the  same  approach  as  traditional  benchmarking,  but  the  cost  of  this 
approach  is  modest  and  works  well  considering  the  resistance  to  traditional  benchmarking  in 
the  DoD  acquisition  context. 

Five  steps  are  required  to  prepare  a  should-cost  proposal  using  parametric 


estimation  tools. 

Step  1: 

Develop  a  detailed  understanding  of  the  proposer’s  estimate.  Include 
product  scope,  architecture,  and  methods  of  development  by 
reviewing  the  proposal  and  proposer’s  basis  of  estimate. 

Step  2: 

Use  a  parametric  estimation  tool  to  develop  an  estimate  that  matches 
the  proposer’s  estimate  as  closely  as  possible.  Estimates  of  size 
must  match  exactly. 

Step  3: 

Perform  a  sensitivity  analysis  to  identify  the  productivity  factors  having 
the  greatest  effects  on  program  performance. 

Step  4: 

Prepare  an  alternative  estimate  with  the  adjusted  parameters. 

Develop  a  briefing  demonstrating  the  changed  parameters  and  new 
estimate. 
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Step  5:  Conduct  a  workshop  to  help  the  contractor  plan  potential  performance 
improvements. 

Step  1:  Develop  a  Detailed  Understanding  of  the  Proposer’s  Estimate 

This  step  will  require  access  to  many  details  of  the  contractor’s  basis  of  estimate  and 
some  interviews  with  the  contractor’s  staff.  This  step  requires  access  to  the  program 
management  and  engineering  staff  who  provided  the  size,  product  complexity,  and  project 
environment  factors  used  for  the  estimate.  Usually,  the  interviews  will  require  a  full  day  and 
may  require  an  additional  phone  call  to  understand  the  contractor’s  meaning  and  intent  for 
some  data.  Analyzing  the  basis  of  estimate  may  require  as  much  as  five  to  seven  days  in 
total.  Understanding  the  scope  of  work  and  complexity  of  the  proposed  product  is  not  easy 
since  the  WBS  (e.g.,  task  sheets)  structure  of  the  proposal  may  cause  parts  of  the  estimate 
to  be  represented  in  several  different  sections. 

•  Begin  preparation  by  reviewing  product  requirements,  including  proposed 
product  architecture.  Identification  of  complexity  factors  such  as  aggressive 
key  performance  measures,  safety,  interfaces,  and  others  will  be  essential  to 
preparing  the  estimate. 

•  Provide  the  contractor  with  requirements  for  data  and  interviews. 

With  the  contractor,  complete  the  following: 

•  Review  analogies  used  for  developing  the  size  estimate.  Did  setting  the  size 
follow  a  standard  procedure  used  previously  by  the  company?  Is  there  any 
reason  the  size  would  have  been  adjusted  to  meet  a  target  price?  Use  these 
factors  to  set  a  potential  range  for  the  size  estimate. 

•  Check  the  scope  definition  to  see  which  components  and  work  products  will 
be  delivered  and  to  whom  they  will  be  delivered.  Count  every  delivery  outside 
the  development  team  (e.g.,  product  certification  and  public  demonstration). 

•  Check  the  domain  definition  and  whether  the  product  is  considered  to  be  new 
or  a  modification  and  enhancement. 

•  Identify  the  collection  of  task  sheets  representing  the  WBS  that  will  be  utilized 
by  the  estimation  tool.  Sum  up  the  efforts  on  these  task  sheets  that 
correspond  to  the  estimation  tool  outputs. 

•  Review  the  definition  and  computation  of  application  complexity.  Specifically 
look  for  performance  criteria  and  quality  attributes  that  may  represent  specific 
baseline  attributes  in  the  estimation  tool  knowledge  base.  This  step  is 
important  because  there  may  be  inconsistencies  between  the  proposer’s  use 
of  terminology  and  the  tool’s  knowledge  base  use  of  the  same  terms.  For 
instance,  some  performance  requirements  might  use  the  phrase  “real  time”  to 
mean  “very  fast”  where  the  normal  interpretation  is  “deadline  driven.” 

•  Review  “Manager’s  Checklist  for  Validating  Software  Cost  and  Schedule 
Estimates”  (Park,  1994)  to  confirm  satisfaction  with  the  contractor’s 
estimation  process  and  resulting  basis  of  estimate. 

•  Document  the  size  estimate  and  the  knowledge  base  factors  to  be  applied  for 
each  component  that  will  be  estimated.  The  size  values  should  be  the  current 
baseline  product,  proposed  reuse,  modification,  and  new  development.  Use 
of  proxy  measures  such  as  ESLOC  will  add  uncertainty  to  the  estimate. 
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At  the  completion  of  this  step,  you  should  be  ready  to  supply  the  parametric  inputs  in 
the  next  step. 

Step  2:  Match  Proposer’s  Estimate 

The  purpose  of  this  step  is  to  use  the  parametric  tool  to  produce  an  estimate  that 
matches  the  contractor’s  estimate  as  closely  as  possible.  The  estimates  should  match, 
within  a  small  difference  on  size,  effort,  schedule,  and  defects.  Many  different  parameters 
must  be  tested  to  achieve  a  satisfactory  result. 

Perform  the  following  activities  during  this  step: 

•  Clearly  identify  as  much  of  product  context  as  the  tool  allows.  Most  tools 
allow  specification  of  product  domain  (e.g.,  avionics),  development 
methodology,  and  development  language. 

•  Begin  by  entering  base,  new,  modified,  and  deleted  size  estimates.  ESLOC 
can  be  used  as  a  last  resort,  but  this  increases  the  uncertainty  in  the 
estimate.  It  is  not  possible  to  use  an  ESLOC  value  to  back  out  the  base,  new, 
modified,  and  deleted  values. 

•  Record  additional  estimation  tool  parameter  values  such  as 

o  available  tools  and  platforms, 

o  experience  of  team  members  in  both  development  and  architecture, 
o  organizational  process  maturity, 
o  quality  assurance  and  testing,  and 

o  factors  affecting  team  performance,  such  as  cohesion  and 
geographical  proximity. 

Detailed  familiarity  with  the  parametric  tool  is  required  for  this  step.  DoD 
contractors  are  and  will  claim  to  be  high-caliber  development  organizations. 
Interviews  are  a  good  mechanism  for  obtaining  the  parameter  values,  but 
experience  and  judgment  are  necessary  for  trustworthy  results. 

•  Modify  the  parameter  values  of  the  baseline  to  match  the  contractor  estimate. 
This  step  may  be  difficult  and  tedious.  Even  a  fairly  simple  tool  like  COCOMO 
II  has  22  factors  affecting  productivity  plus  various  sizing  factors.  Once  the 
initial  estimate  is  prepared  with  contractor  sizing  and  product  domain 
information,  it  is  time  to  match  the  contractor  estimate  by  adjusting  quality 
and  productivity  parameters. 

•  Save  the  matched  estimate  as  a  baseline. 

If  no  reasonable  match  can  be  made,  then  it  is  time  to  re-check  the  Park  (1994)  checklist 
and  re-interview  the  contractor.  Most  likely,  there  is  a  misinterpretation  of  some  size 
measure,  knowledge  base  parameter,  or  performance  parameter.  It  is  also  possible  that  the 
contractor’s  WBS  has  been  misinterpreted. 

Step  3:  Perform  Sensitivity  Analysis 

The  sensitivity  analysis  is  necessary  in  order  to  make  concrete  suggestions  about 
productivity  improvements.  Productivity  parameters  will  include  such  factors  as  team 
cohesion,  developer  experience,  project  environment,  and  process  maturity.  Product  quality 
parameters  will  address  questions  about  the  target  environment,  testing,  and  stability  of  the 
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specification.  Parameters  affecting  product  quality  should  generally  be  excluded  from  the 
sensitivity  analysis  unless  some  error  has  been  identified  in  the  proposal. 

•  If  the  tool  provides  a  sensitivity  analysis,  then  use  the  suggested  top  10 
parameters  for  improvement  potential.  If  the  tool  lacks  this  capability,  it  may 
be  necessary  to  apply  brute  force  or  Monte  Carlo  methods  to  determine  the 
parameter  sensitivity. 

•  List  the  parameters  to  be  tested  for  alternative  estimates. 

Step  4:  Prepare  Alternative  Estimates 

•  Re-run  estimates  with  the  identified  performance  criteria  set  to  revised 
values.  The  revised  values  are  selected  from  benchmark  data.  These  values 
may  be  taken  from  the  best  projects  in  the  tool  vendor’s  database  or  another 
source. 

•  Document  the  alternative  schedule,  effort,  and  defects  along  with  the  revised 
resource  allocation  (how  much  effort  is  suggested  for  top  few  roles). 

•  Save  the  new  baselines  with  identification. 

•  Document  the  changes  to  the  affected  parameters. 

•  Document  the  differences  from  the  contractor’s  baseline  in  schedule,  effort, 
defects,  and  cost. 

•  Run  a  second  sensitivity  analysis.  If  the  sensitivity  analysis  suggests 
significant  additional  improvements  are  possible,  then  repeat  this  step  and 
develop  a  second  should-cost  estimate  and  proposal. 

Summarize  the  results  in  a  briefing  making  comparisons  of  estimated  results  and 
alternative  parameter  values.  Associated  with  each  alternative  should  be  a  discussion  of  the 
rationale  for  the  potential  improvements  and  how  they  might  be  achieved.  If  more  than  one 
estimate  will  be  presented,  then  be  prepared  to  discuss  the  relative  improvement  achieved 
by  each. 

Step  5:  Workshop 

The  workshop  begins  with  a  presentation  of  the  analytical  results  and  concludes  with 
some  recommendations  for  action.  A  workshop  is  necessary  as  the  contractor  must  agree  to 
planning  and  resourcing  to  make  changes. 

•  Display  the  baseline  estimate  beginning  with  the  usual  values:  size,  effort, 
schedule,  and  defects. 

•  Show  the  sensitivity  analysis  used  to  arrive  at  the  new  estimation  parameter 
values. 

•  Provide  the  actual  list  of  parameter  values  applied  for  the  new  estimate. 

•  Display  the  revised  estimate  showing  the  comparison  of  the  values  to  the 
baseline. 

•  Provide  comparisons  and  explanations  of  initial  and  revised  parameter 
values. 

•  Allow  contractor  evaluation  of  potential  for  change. 

•  Achieve  agreement  on  action  items  to  resource  changes. 
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Results 

The  SEI  participated  with  AT  Kearney  (ATK)  in  a  should-cost  analysis  for  the  F-22 
3.2a  contract.  The  SEI  used  the  method  described  here  and  ATK  applied  bottom-up 
analysis.  Both  approaches  led  to  very  similar  cost  savings,  which  gave  the  resulting 
recommendations  very  strong  weight.  As  a  result  of  this  should-cost  effort,  the  program 
office  was  able  to  negotiate  a  15%  reduction,  $32  million  cost  savings.  These  results  were 
reported  in  a  recent  OSD  (2012)  publication  Better  Buying  Power  II. 

There  were  several  lessons  learned  during  this  effort.  Many  of  the  lessons 
correspond  to  the  recommendations  in  the  aforementioned  RAND  report. 

1 .  A  dedicated  independent  team  is  needed.  This  team  was  focused  on  the 
should-cost  effort  and  not  distracted  by  contracting  and  immediate  technical 
problems. 

2.  Use  of  multiple  methods  for  should-cost  has  value  to  program.  The  methods 
used  by  ATK  and  SEI  were  independent  and  different.  The  results  were 
similar  and  carried  a  great  deal  of  weight  in  negotiations  because  of  the 
independence. 

3.  A  contractor’s  estimation  procedure  based  solely  on  historical  data  is 
insufficient.  Such  contractors’  estimates  may  be  defensible  but  miss  the 
opportunity  for  benchmarking  against  competition  and  industry-wide 
comparisons.  Should-cost  is  a  method  that  requires  available  benchmarks  for 
both  cost  and  quality  and  specifically  identifies  the  driving  factors  behind  cost 
and  quality. 

4.  The  contractors’  usage  of  estimation  tools  must  be  examined  carefully. 
Contractors  may  change  the  cost  estimation  tool’s  baseline  data  in  order  to 
match  contractor  performance  history.  This  approach  can  compromise  the 
ability  to  use  the  parametric  model  as  a  baseline.  Using  the  parametric  model 
as  a  benchmark  required  significant  analysis  to  arrive  at  a  baseline  value  that 
matched  the  contractor’s.  Contractors  had  misinterpreted  some  input 
productivity  factors  and  adjusted  the  output  calculations  instead. 

5.  Not  all  parameters  are  easy  to  identify.  For  example,  SEER  makes  use  of  a 
parameter  that  can  be  used  to  account  for  independent  development  teams 
when  size  has  not  been  partitioned  to  the  component  level  in  the  estimate. 
Partitioning  the  work  allows  for  a  more  aggressive  schedule  estimate  since 
teams  are  able  to  operate  independently  until  integration  testing.  This  may  be 
difficult  to  detect  from  the  available  documentation. 

6.  Consider  the  effects  of  adding  automation  or  tooling  to  testing  and  other 
process  changes.  Cost  savings  are  often  made  possible  by  making  process 
changes;  however,  process  changes  can  take  time  to  execute.  Some  savings 
that  were  suggested  in  the  F-22  analysis  were  not  achievable  within  the  time 
horizon  of  the  3.2a  effort.  Recommendations  will  be  accepted  or  rejected  as 
part  of  the  negotiation  process. 

There  were  a  number  of  reasons  to  consider  the  F-22  analysis  a  success.  The 
government  certainly  was  happy  to  negotiate  a  better  price.  Even  though  some  of  the  work 
between  analysts  and  contractors  was  contentious,  the  contractors  were  able  to  agree  to  a 
number  of  suggested  improvements.  An  additional  should-cost  analysis  was  also  conducted 
for  the  next  contract  block.  The  second  time  through  there  was  already  evidence  of 
improved  performance  and  much  less  contention  during  the  analysis. 
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It  will  be  a  while  before  the  final  numbers  are  available  from  the  F-22  modernization 
work.  Hopefully,  that  will  also  be  a  success  story. 

References 

Boito,  M.  (2012).  The  Air  Force’s  experience  with  should-cost  reviews  and  options  for  enhancing  its 
capability  to  conduct  them.  Retrieved  from 

http://www.rand.org/pubs/technical_reports/TR1 1 84.html 

Camp,  R.  C.  (1989).  Benchmarking:  The  search  for  industry  best  practices  that  lead  to  superior 
performance.  Asq  Press. 

Carter,  A.  (2010,  June).  Better  buying  power.  Retrieved  from 

http://www.acq.  osd.mil/docs/USD(AT&L)_Memo_to_Acquisition_Professionals_June_28_2010.p 
df 

Office  of  the  Secretary  of  Defense.  (2012).  Better  buying  power  II.  Retrieved  from 

http://www.acq.osd.mil/docs/BBP  Fact  Sheet  (13  NOV)  Final.pdf 

Park,  R.  (1 995).  Manager’s  checklist  for  validating  software  cost  and  schedule  estimates,  A 
(CMU/SEI-95-SR-004).  Retrieved  from  Carnegie  Mellon  University,  Software  Engineering 
Institute  website:  http://www.sei.cmu.edu/library/abstracts/reports/95sr004.cfm 

Womack,  J.  P.,  Jones,  D.T.,  &  Roos,  D.  (1990).  The  machine  that  changed  the  world.  Scribner. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-406- 


*  * 


iNPS, 


,  PRAtSU(«lAPmci£NrMM  ( 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CA  93943 


www.acquisitionresearch.net 


